System and method of personalized navigation inside a business enterprise

ABSTRACT

Systems and methods for tracking movement of individuals through a building receive, by one or more RF nodes disposed near an entrance to the building, RF signals from RF-transmitting mobile devices carried by persons near the entrance, capture an image of the persons while they are near the entrance, determine an identity and relative distance of each RF-transmitting mobile device from each RF node based on information associated with the RF signals received by that RF node, detect humans in the image, determine a relative depth of each human in the image, and assign the identity of each RF-transmitting mobile device to one of the humans detected in the image based on the relative distance of each RF-transmitting mobile device from each RF node and the relative depth of each human in the image, thereby identifying each individual who to be tracked optically as that individual moves throughout the building.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 17/307,254, filed May 4, 2021, titled “System andMethod of Personalized Navigation Inside a Business Enterprise,” whichis a continuation application of U.S. patent application Ser. No.16/857,399, filed Apr. 24, 2020, issued as U.S. Pat. No. 11,022,443,titled “System and Method of Personalized Navigation Inside a BusinessEnterprise,” which is a continuation application of U.S. patentapplication Ser. No. 16/658,951, filed Oct. 21, 2019, issued as U.S.Pat. No. 10,634,506, titled “System and Method of PersonalizedNavigation Inside a Business Enterprise,” which is acontinuation-in-part application of U.S. patent application Ser. No.16/163,708, filed Oct. 18, 2018, issued as U.S. Pat. No. 10,455,364,titled “System and Method of Personalized Navigation Inside a BusinessEnterprise,” which is a continuation-in-part application of U.S. patentapplication Ser. No. 15/839,298, filed Dec. 12, 2017, issued as U.S.Pat. No. 10,634,503, titled “System and Method of PersonalizedNavigation Inside a Business Enterprise,” which claims the benefit ofand priority to under 35 U.S.C. § 119(e) of U.S. Provisional ApplicationNo. 62/432,876 titled “System and Method of Personalized NavigationInside a Business Enterprise,” filed on Dec. 12, 2016, the entireties ofwhich patent applications and provisional application are hereinincorporated by reference for all purposes.

FIELD OF THE INVENTION

The invention relates generally to systems and methods for providingpersonalized navigation for people while inside certain businessenterprises.

BACKGROUND

A common complaint among shoppers is that they are often frustrated bynot knowing where certain items are located within the store. Theywander about inefficiently through the aisles searching for items ontheir shopping list, often retracing steps, taking the long path intheir quest of the desired items.

SUMMARY

According to one embodiment, the invention relates to a system fortracking locations of individuals in a building. The system comprises atleast one radiofrequency (RF) node disposed near an entrance to thebuilding. The at least one RF node has an RF receiver to receive RFsignals from RF-transmitting devices near the entrance to the building.At least one optical device disposed near the entrance to the building,the at least one optical device capturing an image of a plurality ofpersons while the plurality of persons is near the entrance to thebuilding. A controller is in communication with the at least one RF nodeto obtain therefrom information associated with the RF signals receivedby the RF receiver of that at least one RF node and in communicationwith the at least one optical device to obtain therefrom the capturedimage. The controller is configured to determine an identity of eachRF-transmitting device and an angular position of that RF-transmittingdevice with respect to each RF node of the at least one RF node based onthe information associated with the RF signals obtained by thecontroller from that RF node of the at least one RF node. The controlleris further configured to detect an orientation marker and a plurality ofhumans in the image obtained by the controller from the at least oneoptical device and to assign the identity of each RF-transmitting deviceto one of the plurality of humans detected in the image based on aposition of the orientation marker in the image relative to each humandetected in the image and on the determined angular position of thatRF-transmitting device with respect to each RF node of the at least oneRF node. Thereby, each individual is identified who is to be locatedoptically as that individual moves throughout the building.

In one example embodiment, the at least one RF node is configured todetermine a relative signal strength indicator (RSSI) value for the RFsignals received from each RF-transmitting device. The informationobtained by the controller from the at least one RF node includes theRSSI values. The controller is configured to estimate a distance of eachRF-transmitting device from each RF node based on the RSSI values forthe RF signals received by that RF node from that RF-transmitting deviceand to use the estimated distance of each RF-transmitting device fromeach RF node when assigning the identity of each RF-transmitting deviceto one of the plurality of humans detected in the image.

In another example embodiment, the controller is further configured todetect in the image a mobile phone held by one of the plurality ofhumans detected in the image and to assign the identity of eachRF-transmitting device to one of the plurality of humans detected in thecaptured image based on the which of the plurality of humans detected inthe image is holding the detected mobile phone.

In another example embodiment, the controller is further configured toarrange, for each RF node, the RF-transmitting mobile devices in anangular order based on the angular positions of the RF-transmittingmobile devices from that RF node, arrange the humans captured in theimage into an angular order based on relative positions of the pluralityof humans detected in the captured image, and compare the angular orderof humans detected in the captured image with the angular order of theRF-transmitting mobile devices when assigning the identity of eachRF-transmitting mobile device to one of the plurality of humans detectedin the captured image. The controller may be further configured toarrange the angular order of the plurality of humans in the capturedimage to be in respect to the at least one RF node before the angularorder of humans detected in the image is compared with the angular orderof the RF-transmitting mobile devices.

In yet another example embodiment, the system further comprises aplurality of optical devices disposed throughout the building, andwherein the controller is further configured to optically track eachidentified individual through the building based on detecting thatidentified individual in images received from at least some of theplurality of optical devices over time.

In yet another example embodiment, the information carried by RF signalstransmitted by a given RF-transmitting mobile device includes a shoppinglist. The controller may be further configured to determine a routethrough the building based on items on the shopping list and transmitthe route to the given RF-transmitting mobile device for display on ascreen of the given RF-transmitting mobile device.

As another example, the system further comprises a plurality of opticaldevices disposed throughout the building, and the controller is furtherconfigured to optically track each identified individual through thebuilding based on detecting that identified individual in imagesreceived from at least some of the plurality of optical devices overtime.

In other examples, the information carried by RF signals transmitted bya given RF-transmitting mobile device includes a shopping list. Thecontroller may be further configured to determine a route through thebuilding based on items on the shopping list and transmit the route tothe given RF-transmitting mobile device for display on a screen of thegiven RF-transmitting mobile device. The at least one RF node maycomprise two or more RF nodes.

In another example, the controller is further configured to compute alocation for each RF-transmitting mobile device based on the angularpositions of the RF-transmitting mobile devices received from the atleast one RF node, to arrange the RF-transmitting mobile devices in adepth order based on the computed locations of the RF-transmittingmobile devices with respect to the at least one RF node, to arrange thehumans captured in the image into a depth order based on relativepositions of the plurality of humans detected in the captured image, andto assign the identity of each RF-transmitting mobile device to one ofthe plurality of humans detected in the captured image by comparing thedepth order of humans detected in the image with the depth order of theRF-transmitting mobile devices.

According to another embodiment, the invention relates to a method fortracking locations of individuals in a building. The method comprisesreceiving, by at least one radiofrequency (RF) node disposed near anentrance to the building, RF signals from RF-transmitting mobile devicescarried by a plurality of persons near the entrance to the building. Animage is captured of the plurality of persons while the plurality ofpersons is near the entrance to the building. An identity of eachRF-transmitting mobile device and an angular position of eachRF-transmitting mobile device from each RF node are determined based oninformation associated with the RF signals received by the at least oneRF node. A plurality of humans and an orientation marker are detected inthe captured image. The identity of each RF-transmitting mobile deviceis assigned to one of the plurality of humans detected in the capturedimage based on a position of the orientation marker in the imagerelative to each human detected in the image and on the determinedangular position of each RF-transmitting mobile device with respect toeach RF node, thereby identifying each individual who is to be locatedoptically as that individual moves throughout the building.

In one example embodiment, a relative signal strength indicator (RSSI)value is determined for the RF signals received from eachRF-transmitting device, an estimated distance of each RF-transmittingdevice from each RF node is computed based on the RSSI values for the RFsignals received by that RF node from that RF-transmitting device, andthe estimated distance of each RF-transmitting device from each RF nodeis used when assigning the identity of each RF-transmitting device toone of the plurality of humans detected in the image.

In one example embodiment, a mobile phone held by one of the pluralityof humans is detected in the image, and assigning the identity of eachRF-transmitting device to one of the plurality of humans detected in thecaptured image is based on the which human detected in the image isholding the detected mobile phone.

In another example embodiment, the RF-transmitting mobile devices arearranged in an angular order based on the angular positions of theRF-transmitting mobile devices with respect to the at least one RF node,and the humans captured in the image are arranged into an angular orderbased on relative positions of the plurality of humans detected in thecaptured image, and assigning the identity of each RF-transmittingmobile device to one of the plurality of humans detected in the capturedimage includes comparing the angular order of humans detected in theimage with the angular order of the RF-transmitting mobile devices. Theangular order of humans in the captured image may be rearranged to be inrespect to the at least one RF node before the angular order of humansdetected in the captured image is compared with the angular order of theRF-transmitting mobile devices.

In another example embodiment, a location of each identified individualis optically tracked through the building by comparing successive imagesthat capture that identified individual.

In still another example embodiment, the information associated with theRF signals transmitted by a given RF-transmitting mobile device includesa shopping list. A route through the building may be determined based onitems on the shopping list, and the route may be transmitted to thegiven RF-transmitting mobile device for display on a screen of the givenRF-transmitting mobile device.

In yet another example embodiment, radio signals carrying a currentlocation of a given identified individual within the building may aretransmitted from the at least one RF node to the RF-transmitting mobiledevice carried by that given identified individual.

In still another example embodiment, a location for each RF-transmittingmobile device is computed based on the angular positions of theRF-transmitting mobile devices received from the at least one RF node.The RF-transmitting mobile devices are arranged in a depth order basedon the computed locations of the RF-transmitting mobile devices withrespect to the at least one RF node. The humans detected in the capturedimage are arranged into a depth order based on relative positions of thehumans detected in the captured image. Assigning the identity of eachRF-transmitting mobile device to one of the plurality of humans detectedin the captured image includes comparing the depth order of humansdetected in the image with the depth order of the RF-transmitting mobiledevices.

According to another embodiment, the invention relates to a system fortracking locations of individuals in a building. The system comprises atleast one radiofrequency (RF) node disposed near an entrance to thebuilding. The at least one RF node has an RF receiver channel to receiveRF signals from a plurality of RF-transmitting devices near the entranceto the building. The system further comprises at least one cameradisposed near the entrance to the building. The at least one cameracaptures an image of a plurality of persons while the plurality ofpersons is near the entrance to the building. A controller of the systemis in communication with the at least one RF node to obtain therefrominformation associated with the RF signals received by the RF receiverchannel of that at least one RF node and in communication with the atleast one camera to obtain therefrom the captured image. The controlleris configured to determine an identity of each RF-transmitting deviceand a position of that RF-transmitting device relative to each RF nodeof the at least one RF node based on the information associated with theRF signals obtained by the controller from that RF node of the at leastone RF node, to detect a plurality of humans in the image obtained bythe controller from the at least one camera, to detect an orientationmarker in the image, and to assign the identity of each RF-transmittingdevice to one of the plurality of humans detected in the image based ona position of the orientation marker in the image relative to each humandetected in the image and on the determined position of thatRF-transmitting device relative to each RF node of the at least one RFnode.

Still other aspects, embodiments, and advantages of these exemplaryaspects and embodiments are discussed in detail below. Embodimentsdisclosed herein may be combined with other embodiments in any mannerconsistent with at least one of the principles disclosed herein, andreferences to “an example”, “an embodiment,” “some embodiments,” “analternate embodiment,” “various embodiments,” “one embodiment” or thelike are not necessarily mutually exclusive and are intended to indicatethat a particular feature, structure, or characteristic described may beincluded in at least one embodiment. The appearances of such termsherein are not necessarily all referring to the same embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of at least one embodiment are discussed below withreference to the accompanying figures, which are not intended to bedrawn to scale. The figures are included to provide illustration and afurther understanding of the various aspects and embodiments and areincorporated in and constitute a part of this specification but are notintended as a definition of the limits of the invention. In the figures,each identical or nearly identical component that is illustrated invarious figures is represented by a like numeral. For purposes ofclarity, not every component may be labeled in every figure. In thefigures:

FIG. 1 is a block diagram of an example of a personalized navigationsystem according to certain aspects of the present invention;

FIG. 2 is a diagram of an example a floor plan of a business enterpriseconfigured with a personalized navigation system according to certainaspects of the present invention;

FIG. 3 is an example display of the floor plan of the businessenterprise as it may appear on the screen of the mobile device carriedby the shopper according to certain aspects of the present invention;and

FIG. 4 is a diagram illustrating an example of a system including ashelf camera and an aisle camera tracking a shopper in front of a retailshelf within the business enterprise, according to aspects of thepresent invention.

FIG. 5 is a diagram illustrating an example of a personalized navigationsystem configured to distinguish among multiple persons disposed at theentrance of the business enterprise.

FIG. 6 is a flow diagram of an embodiment of a process fordistinguishing among multiple persons located at the entrance of thebusiness enterprise.

FIG. 7 is a flow diagram of another embodiment of a process fordistinguishing among multiple persons located at the entrance of thebusiness enterprise.

DETAILED DESCRIPTION

Personalized navigation systems according to certain aspects andembodiments use a combination of radiofrequency (RF) technology andoptical imaging technology to identify and track persons at businessenterprises, and software to provide the persons with individualizedinformation and an individualized navigation experience within thebusiness enterprise. As discussed in more detail below, a person canprovide the personalized navigation system with information, such as ashopping list or service request, for example, and receive personalizednavigation or other information in response to aid the person inefficiently completing their objective(s) at the business enterprise. Inthis manner, the person's experience at the business enterprise and canbe improved.

According to one embodiment, a personalized navigation system uses RFtechnology to initially identify a shopper who approaches an entrance tothe business enterprise and uses optical technology to detect and trackmovement of the shopper after the shopper arrives at and enters thebusiness enterprise. To cooperate with the navigation system, theshopper carries a mobile device (e.g., a smartphone or smart watch) withRF transmitting and RF receiving capability. In certain embodiments, themobile device runs certain application software that transmits RFsignals containing an identity of the shopper and the shopper's shoppinglist. The shopper can acquire the application software and download itto the mobile device from an “app store”. Many business enterprises arecurrently equipped with RF transmitters, RF receivers, and videocameras, and advantageously, the navigation systems described herein donot require any hardware modifications to this existing RF and videoequipment.

FIG. 1 shows an example of a personalized navigation system 100configured to operate in a business enterprise 110 with at least one RFnode 120. Any number of RF nodes, designated 120-1 to 120-n (n being apositive integer number), may be included. Each RF node 120-1 to 120-nincludes a corresponding RF receiver antenna 130-1, 130-n (generally,130). Optionally, each RF node 120 can also have an RF transmit antenna(not shown). Each RF node 120 can be placed at or near an entrance tothe business enterprise 110. Examples of the business enterprise 110include, but are not limited to, grocery stores, supermarkets,department stores, hardware stores, warehouses, and retail stores. Ingeneral, the location of the RF node 120 near the entrance of thebusiness enterprise 110 facilitates detection of an RF-transmittingmobile device 140 associated with a person approaching the entrance,provided that the mobile device 140 is running application software thatprovides personalized navigation of the business enterprise, asdescribed herein. The mobile device 140 engages in communication withone or more of RF nodes 120 using a wireless communication technology,such as Bluetooth®, Wi-Fi, or Near Field Communication (NFC). If thebusiness enterprise 110 has multiple entrances, with at least one RFnode 120 disposed near each entrance, the personalized navigation system100 can be aware of which entrance the person is entering based on whichRF node 120 detects the mobile device 140.

During typical operation, a person with the mobile device 140 approachesan entrance to the business enterprise (i.e., a building) 110. Themobile device 140 runs a personalized navigation app and transmits RFsignals. In certain examples the RF signals carry an identifierassociated with the person by which an operator of the businessenterprise 110 knows the person. For example, the identifier may includethe person's name, a telephone number, a rewards program numberconnected with the business enterprise, or other identifyinginformation. The RF signals may also carry the person's shopping listidentifying those items that the person wishes to find upon visiting thebusiness enterprise 110. Typically, the person may prepare this shoppinglist before visiting the business enterprise 110; however, the shoppinglist can be constructed or edited at any time before or after the personarrives at the business enterprise 110.

When the person comes into range of an RF receiver antenna 130, themobile device 140 establishes communications with the RF node 120. Inparticular, in certain examples the mobile device 140 may pass to the RFnode 120 the identifier and shopping list. The mobile device may also oralternatively pass other data to the RF node 120, such as a set ofinstructions or other information for a technician or similar repairstaff performing certain services at the business enterprise 110, forexample. In certain examples, the RF node 120 forwards the identifierand shopping list or other data to a computer processing unit (alsocalled a controller) 150, which can use this identifier to access adatabase 160 where information relating to the person associated withthe identifier is kept. This information can include records of priorvisits to the business enterprise 110 by the person, for example.Although the computer processing unit 150 and database 160 are shown inFIG. 1 to be within the business enterprise 110, they may reside insteadat other sites, such as on a third-party network of computers andservers referred to broadly as “the cloud.”

FIG. 2 shows a simple example of a floor plan 200 of the businessenterprise 100 with various aisles 210-1 through 210-7 (generally, 210)and video cameras 220-1 through 220-n (generally 220) placed throughoutthe store, preferably to provide full coverage of the interior of thebusiness enterprise 100, wherever visitors may walk. The businessenterprise 100 can also include one or more RF transmitters 230 and anetworking device 240. Each video camera 220, each RF node 120, and eachRF transmitter 230 is in communication with the networking device 240 bya wireless or wired communication link (not shown). The networkingdevice 240 is in communication with the computer processing unit 150(shown in FIG. 1 as discussed above), which in one embodiment resides onthe cloud 250. A wired or wireless communication link 260 connects thenetworking device 240 to the central processing unit 150.

As discussed above, tracking of persons within the business enterprise110 can be accomplished using optical technology; in particular, bycapturing and processing images from the video cameras 220 locatedthroughout the business enterprise 110. According to one embodiment,during typical operation of the personalized navigation system, thevideo cameras 220 continuously capture images within their fields ofview. At least one video camera 220 can be placed proximate eachentrance of the business enterprise 110 to acquire images of personsentering the business enterprise. In some embodiments, multiple videocameras 220 can be placed proximate each entrance in such a way as toprovide a complete field of view, or at least a functionally sufficientfield of view, of the area around the entrance such that images of allpersons entering the business enterprise 110 can be acquired. Asdiscussed above, when a person having a mobile device 140 configured torun the application software to engage with the personalized navigationsystem 110 (referred to as a tracked person) arrives at an entrance tothe business enterprise 110 the RF node 120 at that entrance receivesthe identifier, and optionally other information (such as the shoppinglist), from the mobile device 140. At the same time, the video camera(s)220 proximate that entrance capture images of the area around theentrance, and at least some of these images should contain the trackedperson. As discussed above, in certain examples, the computer processingunit 150 knows which entrance a person used to enter the enterprise 110based on which RF node 120 detected the person and known locations ofeach RF node. Accordingly, the computer processing unit 150 knows whichvideo camera or cameras 220 are in position to capture images of theperson. These video cameras 220 pass their captured images to thenetworking device 240, which sends the captured images to the centralprocessing unit 150. The central processing unit 150 includes an imageprocessor that performs image processing techniques adapted to detect aperson within the image and to associate the detected person with themost recently acquired identifier and shopping list.

Techniques for processing images to identify a person within the imagesare known in the art, and any such image processing techniques can beimplemented by the central processing unit 150. For example, the imageprocessor can be adapted to examine images captured by the video camera220-1 positioned at the relevant entrance for a smartphone in the handof an individual, which may indicate that the individual is engagingwith the personalized navigation system 100. Alternatively, or inconjunction, the image processor can be adapted to examine the capturedimages for the head or hands of a person. Since the central processingunit 150 expects the next person to fall within the field of view of thevideo camera 220-1 located at the entrance to be the same as the personwho communicated with the RF node 120-1 located at that entrance, thatdetected person becomes associated with received identifier and shoppinglist. Once a person has been identified in an image and associated withthe received identifier, the personalized navigation system 100 tracksand guides the person as he or she moves through the business enterprise110.

Tracking can be accomplished by collecting images from the various videocameras 220 located amongst the aisles 210 and processing these imagesto follow the tracked person. In certain examples the central processingunit 150 follows the movement of the tracked person as her or she movesfrom one camera field of view to another, dynamically registering andupdating the location of the person within the business enterprise 110.In one example the video cameras 120 operate in parallel, with all orsome of the video cameras providing images to the central processingunit simultaneously. The images can be merged into a map or layout ofthe business enterprise 110, such as shown in FIG. 2 . In some examplestwo or more of the video cameras 220 can have at least partiallyoverlapping fields of view, and in other examples different videocameras 220 are used to monitor different areas of the businessenterprise 110. The video cameras 220 may capture images with differentperspectives. The central processing unit may flatten the images byremoving perspective distortion in each of them, and merges theresulting corrected images into the map. Image stitching techniques canbe used to merge the images.

In certain examples, an image stitching process first performs imagealignment using algorithms that can discover the relationships amongimages with varying degrees of overlap. These algorithms are suited forapplications such as video stabilization, summarization, and thecreation of panoramic mosaics, which can be used in the images takenfrom the cameras 220. After alignment is complete, image-stitchingalgorithms take the estimates produced by such algorithms and blend theimages in a seamless manner, while taking care of potential problems,such as blurring or ghosting caused by parallax and scene movement aswell as varying image exposures inside the business enterprise 110.Various image stitching algorithms and processes are known in the art,and any such image processing techniques can be implemented by thecentral processing unit 150.

A handoff can be made when a tracked person moves from one viewpoint toanother or is seen by one camera 220 and not the others. These handoffsmay be made using the images running in parallel on the centralprocessing unit 150, with the tracked person's location and movementdetermined by the central processing unit using whichever camera 220 hasthe best view of the tracked person.

In certain examples, the video cameras 220 can include depth sensors. Insuch examples, the image stitching operation can be omitted, and eachcamera stream data is processed independently for change, persondetection and recognition. Then, the resulting “areas of interest” areconverted to individual point clouds (described further below) andtransformed in to a single common coordinate system. The translation androtation transformations used for this process are based on the positionand orientation of the video cameras (and their associated depthsensors) in relation to one another. In one example, one camera ispicked as the main sensor and all other camera data is transformed intothe main coordinate system, achieving the same end result as the imagestitching procedure, namely, unification of the actual location of thetracked person among sensors.

In some examples the central processing unit 150 may use knowninformation about the floor plan of the business enterprise to assistwith identifying and tracking persons based on the images acquired fromthe video cameras 220. For example, the central processing unit can useknown shapes and positions of shelving along the aisles 210 to providereference points. At times, a tracked person may be occluded in acamera's field of view, for example, by another person, equipment, orshelving. The personalized navigation system 100 can be configured tostore the tracked person's prior-determined position and comparemultiple image frames to relocate the tracked person after a temporaryocclusion. As discussed further below, the personalized navigationsystem 100 can be configured to provide a proposed route for the trackedperson through the business enterprise 110, and therefore the centralprocessing unit can use a predicted future location of the trackedperson to relocate the person after a temporary occlusion.

According to certain embodiments, the central processing unit 150 canrun an image-processing process, optionally supplemented with depthinformation, to track a person as discussed above. A two-dimensional(2D) optical image capture device (i.e., a video camera 220) with asingle aperture is capable of capturing 2D image information on a plane(film, CCD, etc.). To acquire three-dimensional (3D) informationtypically requires acquisition of additional data. Three-dimensionaldata can be acquired using multiple video cameras 220 or by combiningone or more video cameras with one or more depth sensors. The videocameras 220 can utilize visible light, infrared light, or other opticalwavelength ranges. Depth sensors can be based on infrared, laser orother wavelength emitters that transmit light to an object. Depthsensors typically determine the distance to the object from which thelight that is reflected or backscattered. Alternatively, depth sensorscan utilize acoustic signals to determine distance. In one embodiment,depth sensing is integrated into the video cameras 220.

Image frames are acquired from the video cameras 220. A video camerasystem with depth sensing capability typically outputs video (e.g., RGB,CYMG) and depth field information. Video may optionally be encoded to awell-known format, such as MPEG. The optical and depth information arestitched together. Open libraries such as OpenCV or OpenNI (used tocapture depth images) enable the optical and depth information to bestitched together. Alternatively, a user of the personalized navigationsystem 100 may develop customized software for generating 3D informationor object data generated by optical images and depth sensors.

An initial calibration can be performed over multiple image frames todetermine background information both for 2D optical images and thedepth sensing. During the calibration, any motion (e.g., people) isextracted or ignored during background extraction until stablebackground optical (RGB) and depth information can be stored, forexample, in the database 160. Calibration may be performed periodicallyor may be initiated by the personalized navigation system 100, forexample, if errors are detected.

After calibration is complete, the resulting spatial filter masks can beused to extract an “area of interest” for each video camera 220. Forexample, for a video camera located near an entrance to the businessenterprise 110, the area of interest may correspond to the area betweenthe background and a foreground (area where a person is expected to be),so that everything that is not walls, doors, or other infrastructure(for background) and also not a detected person, is ignored. Thisignoring of the background and foreground focuses on data within thedepth threshold of the area of interest being monitored. Alternatively,the “area of interest” can include a different part of the scene, forexample, the foreground in order to see where the person is in laterrecognition steps and can be expanded or contracted as systemrequirements dictate. In general, the area of interest applies to anycut-out of a scene that is to be the focus within which to performperson tracking.

According to certain embodiments, multiple image frames (e.g., N−1 andN) are obtained and compared, and in certain examples the image framescan include depth information in addition to RGB (color) data, asdiscussed above. Image and depth information can be filtered for noiseand then processed to determine if a difference between two framesexists. This can be done with edge detection, threshold and differencealgorithms, or other image processing techniques. In certain examples,information from the depth sensors is also processed to compare imageframes. The system can use changes between image frames, in particular,changes in the position or orientation of a detected person, to trackthe movement of the person. In some embodiments, change detection can belimited to the area of interest to increase processing speed.

In one embodiment, when the area of interest is determined, a “pointcloud” is generated using the video camera's extrinsic and intrinsicparameters through algorithms for “2D to 3D” data representationconversion preformed on the RGB and/or depth images obtained andprocessed through OpenNI and OpenCV. In one embodiment, the Point CloudLibrary may be used. The object shape and location information generatedfrom the Point Cloud Library are used to identify and track a person inthree dimensions using edge detection, color detection, objectrecognition and/or other algorithms for determining the presence of aperson within the scene. If object information is in the shape of ahuman, for example, then the process continues to track the person.However, if the size, shape or other appearance information indicatesthat the object is not a person, subsequent image frames can be analyzeduntil a person is detected. In some examples, images captured by thevideo cameras 220 may include more than one person. Accordingly, theprocess may compare expected features and/or appearance attributes ofthe tracked person with persons detected in the image frames to continueto track the correct person.

As discussed above, the central processing unit 150 can merge theacquired images from the video cameras 220 into a map to be able totrack identified persons as they moved through the business enterprise.In certain examples, the application software running on the mobiledevice 140 can be configured to display the map or a similar map view orvirtual layout of the floor plan of the business enterprise 110, suchthat the tracked person can view their location within the businessenterprise. The central processing unit 150 can send commands to the RFtransmitters 230—by way of the networking device 240—to transmit RFsignals carrying the updated location of the tracked person, which canbe determined using image processing techniques as discussed above. Themobile device 140—with its RF receiver—receives these signals andregisters the updated location of the person within the applicationsoftware, which can show the location of the person within the virtuallayout of the business enterprise 110 displayed on the mobile device140.

FIG. 3 shows an example display 300 of the floor plan of the businessenterprise 110, with aisles 210, as it may appear on the screen of themobile device 140 (FIG. 1 ) carried by the tracked person. In thisexample, an arrow 310 indicates the current location of the trackedperson. As discussed above, the personalized navigation system 100, andthe application software running on the mobile device 140, can beconfigured to guide the tracked person through the business enterprisebased on the information received along with the identifier. Forexample, where a shopping list is provided, the personalized navigationsystem 100 can access information from the business enterprise 110 thatidentifies where in the aisles 210 the items on the shopping list arelocated. In one example, the central processing unit 150 (FIG. 1 , FIG.2 ) examines the shopping list acquired by the initial RF communicationswith the mobile device 140 and accesses a database that stores thelocations of all items within the business enterprise 110. The centralprocessing unit can be further configured to match descriptions of itemson the shopping list with product SKUs or other identifying informationin the database to obtain a location of each item within the businessenterprise 110. The database can be part of the personalized navigationsystem 100, obtained from an operator of the business enterprise 110 andregularly updated, or the database can be maintained by the operator ofthe business enterprise and accessed by the personalized navigationsystem (e.g., by the central processing unit 150) via the cloud 250.

In FIG. 3 , a directional dashed line 320 identifies a proposed routethrough the business enterprise 110. After the central processing unit150 has examined the shopping list and identified where the desireditems 330 on the shopping list are located within the businessenterprise 110, a route 320 is proposed through the business enterprisefor efficiently obtaining the items. In one embodiment, the route isbased on the shortest distance to acquire all the items. Another routecan be to direct the person to non-perishable or unrefrigerated itemsinitially, and leaving perishable or refrigerated items for the end ofthe route. The central processing unit 150 sends the map and itemlocation information to the mobile device 140 via the RF transmitters230 (FIG. 2 ). The application software executing on the mobile device140 displays the route 320 based on the items on the tracked person'sshopping list. To supplement the positioning information and calibrationprovided by the video cameras, the mobile device 140 can have inertialsensors. Techniques for using inertial sensing to enhance positioninginformation are described in U.S. Pat. No. 8,957,812, issued Feb. 17,2015, titled “Position Tracking System and Method using Radio Signalsand Inertial Sensing,” the entirety of which U.S. patent is incorporatedby reference herein. In other examples, additional RF nodes (similar tothe RF nodes 120) with RF transmitting/receiving capability can be usedto supplement the positioning information provided by the video camerasand enhance the tracking capability of the personalized navigationsystem 100.

In addition to identifying the desired items 330, the central processingunit 150 can notify the person of an item that may be of interest, asthe person's current location approaches the location of that item, evenif that item is not on the shopping list. Such an advertisement may bebased on the shopping history of the person, for example. As discussedabove, in certain examples the information provided from the mobiledevice 140 to the RF node 120 (and therefore to the central processingunit 150) can include a service request. Accordingly, in such examplesinstead of or in addition to displaying the locations of the desireditems 330, the location of the service desk or other relevantinformation can be displayed on the map, and the route 320 can beconfigured to guide the person to that location.

Referring to FIG. 4 , another aspect of this disclosure is a producttracking and checkout system 400 that can keep track of products takenby customers as they shop in the business enterprise 110, integratingwith the person's shopping list and also eliminating the need forshoppers to wait in cashier lines. This system 400 enables customers 410to any or all of take products 412, 414 off of the shelves 416, put themin their cart, register the products that are taken and who took theproducts, automatically pay for the products and leave the store withthe products purchases automatically logged by the retail system withouthaving to go through a checkout system. Aspects of this system couldimprove the speed and experience of the shopping process for customersand reduce the cost of cashiers for the retailers.

For example, retailer's costs can be reduced with a system in place thatautomatically keeps track of inventory on shelves and/or what is takenoff these shelves by customers to automatically keep track of whatcustomers take from stores and to manage inventory on shelves. Theability to track inventory and what products customers remove fromshelves can improve the cost basis for retailers by eliminating the needfor cashiers or extra staff to constantly go to shelves to inspect whatitems need replacing and re-stocking. In addition, the system can updatethe shopping list received from a tracked person based on items foundsand taken by the tracked person and update the displayed route 320 basedon the progress made by the tracked person.

It is appreciated that variations of image processing can be used forshelf and product tracking. One aspect of the system 400 includes aproduct recognition camera 420 facing the shelves to view what productsare on the shelf and what products are removed by customers. The systemmay have one or more first shelf facing cameras 420 with a view angle422 focused on the shelf to see what products are there and whatproducts are removed. However, there may also be situations where one ormore shelf focused product recognition cameras 420 may not be sufficientas there may be times that two people reach for products in the samearea, potentially even cross arms while reaching for their individualproducts, and/or possibly blocking the view of the one or more producttracking cameras 420 when reaching and grabbing the product on theshelf.

Thus, an embodiment of the system incorporates an additional outwardlooking (aisle facing) camera 430. Thus, an aspect of this embodiment ofthe system includes at least two cameras on an integrated arm mount 440.At least one first product tracking camera 420 is oriented to focus onthe products on the shelf and at least one second aisle tracking camera430 is oriented to focus on the aisle and the customers doing theshopping. Both cameras can be a video camera, and both cameras can be avideo camera of the video cameras 220-1 through 220-n (generally 220)placed throughout the business enterprise, as discussed above, toprovide full coverage of the interior of the business enterprise 110.Thus, in this embodiment, at least one camera 420 (“shelf trackingcamera”) may be used primarily for product recognition on the shelf andat least one additional camera 430 (“aisle tracking camera”) may be usedprimarily for customer skeletal tracking to confirm where that customeris reaching.

Some advantages of this embodiment of the system 400 are that by usingat least one aisle tracking camera 430 to focus into the aisle and onthe shopper, the system can eliminate any occlusion issues from theshopper standing in front of the shelf-facing camera 420 or any of theother video cameras 220. In addition, the combination of the first shelffacing camera 420 and second aisle facing cameras 430 can also preventthe cameras from confusing what item was taken should two shoppers reachin the same area for products and either cross arms or occlude thecamera potentially causing the system to charge the wrong customer forthe item taken.

Aspects of this embodiment of the system 400 having the dual cameras caninclude accomplishing multiple functions from at least one first camera420 and at least one second camera 430 including shopper registration,shopper movement tracking, and product identification on retailshelving, inventory tracking, and monitoring the amount of products onthe shelving.

Typically, multiple persons may be entering and/or leaving the entranceof an enterprise at a given moment. Further, more than one of them maybe operating an RF-transmitting mobile device at that moment, and an RFnode of the personalized navigation system, located near the entrance,may be in communication with a plurality of them. In addition, otherspassing through or near the entrance at that moment may not be carryinga mobile device or may have their mobile devices are turned off.Personalized navigation systems described herein are configured todistinguish between people participating in the personalized navigationand between participants and non-participants. Unlike participants,non-participants are not interacting through RF communications with thepersonalized navigation system, but like participants, non-participantsmay still be optically tracked through the enterprise.

It is appreciated that aspects of the cameras and the system can alsoinclude color sensing, comparison and depth sensing, which can beaccomplished for example with infrared sensing. The first shelf trackingcamera 420 can use either or both of color and depth sensing to registerthe products' position and to recognize the actual products on theshelf. The second aisle tracking camera 430 can use depth sensing toperform skeletal tracking to confirm where that customer is reaching.Confirmation of which customer selected which product on the shelf isachieved by the shelf camera 420 providing product identification andremoval (from shelf) detection and by the position of the person's armin relation to the item and, upon removal, the item actually in the handof the customer provided by the aisle camera 430. The fusing of thefunctions of these two cameras provides a much more robust method forconfirming what was taken off the shelf and by which shopper.

FIG. 5 shows an embodiment of a personalized navigation system 100configured to distinguish among individuals who pass through an entranceof the business enterprise 110 concurrently or next to each other. Nearthe entrance are located one or more RF nodes 120-1, 120-2 (generally,120) and one or more cameras 220 (only one shown to simplify thedrawing). Each RF node 120 has a corresponding RF receiver antenna130-1, 130-n (generally, 130) and an RF transmitter (not shown). In oneembodiment, each RF node 120 has an antenna array comprised of two ormore antennae (or antenna elements) with known spatial separation(s).Timing differences between the RF signal received at each of the atleast receiver antennae are used by the RF node to estimate angle ofarrival of the RF signal. An example of a multiplexing technique formultiplexing antenna of an antenna array is described in U.S. Pat. No.8,749,433, issued Jun. 10, 2014, titled “Multiplexing Receiver System”,the entirety of which is incorporated by reference herein.

Each RF node 120 and each camera 220 is in a fixed location and incommunication with the controller 150, which knows the relativelocations of each such electronic equipment (i.e., the pan and tiltangles and distance between cameras 220, between RF nodes 120, betweeneach camera 220 and each RF node). Each camera 220 has a field of view510 that covers an area near the enterprise entrance. As previouslydescribed, the controller 150 is configured to detect a person within animage captured by the camera(s) 220 and associate the detected personwith an identifier that was transmitted by the person's mobile deviceand recently received by the RF node.

As illustrated in FIG. 5 , multiple persons 500-1, 500-2, 500-3(generally, 500) are concurrently near the enterprise entrance (whetherentering or leaving). Individuals 500-1 and 500-2 are carryingRF-transmitting mobile devices 140-1, 140-2, respectively. Each mobiledevice 140-1, 140-2 is running application software that providespersonalized navigation of the business enterprise, as previouslydescribed, and is in communication with one or more of RF nodes 120-1,120-2 using a wireless communication technology, such as Bluetooth®,Wi-Fi, or Near Field Communication (NFC). This application software maybe running in the background, that is, the individual does not need todeliberately activate the application to handshake with an RF node; theapplication is in the background broadcasting or listening for joinrequests. By operating the application in the background, the mobiledevice 140 automatically connects with each RF node 120 upon coming intowireless communication range of it. To synchronize communications, eachmobile device 140 handshakes with each RF node 120 with which it is incommunication. In this example, the individual 500-3 is not carrying ornot running a mobile device that is in RF communication with an RF node120. The camera(s) 220 capture images that include all these individuals500-1, 500-2, 500-3; that is, the controller may detect all theindividuals 500-1, 500-2, 500-3 in a single captured image. As describedherein, the controller 150 is configured to determine which receivedidentifier corresponds to which person detected in the captured image,and which detected person does not have or is not using anRF-transmitting mobile device to communicate with the RF node 120.

FIG. 6 shows an embodiment of a process 600 for distinguishing amongmultiple persons located at the entrance of the business enterprise, atleast one of which is carrying and operating an RF-transmitting mobiledevice. For purposes of illustration, consider that the individuals500-2 and 500-3 pass through the entrance side-by-side and continuefurther indoors, whereas individual 500-1 pauses just within theentrance. The mobile devices 140-1, 140-2 of the individuals 500-1,500-2, respectively, communicate (step 602) with the RF nodes 120-1,120-2, sending them their identifiers, and, at about the same time ormomentarily after, the camera 220 captures (step 604) an image with allthree individuals in it.

Each RF node 120 derives (step 606) a received signal strength indicator(RSSI) for the radio signal received from each mobile device 140 andassociates that RSSI measurement with the corresponding identifier. Ingeneral, the closer the mobile device is to the RF node 120, thestronger is the received signal. At step 608, the RF nodes send the RSSImeasurements and associated identifiers, and the camera sends thecaptured image, to the controller. The controller associates thecaptured image with the RSSI measurements and associated identifiersbecause of the near synchronicity of when the camera captured the imageand the RF nodes received the RF signals. The controller also knows therelative locations of the RF nodes to each other and to the camera andtracks which RF node sent which RSSI values.

At step 610, the controller detects the three individuals in the imageand establishes a depth order of the detected individuals with respectto the camera. This determination of depth order may use depthinformation obtained by the camera or by a depth sensor calibrated tothe camera. The controller can rearrange the depth order of humans inthe image from the perspective of or in respect to each RF node, thusfacilitating a comparison of that RF node's distance order, which isbased on its computed RSSI values, with the depth order derived from theimage captured by the camera.

In this example, the controller receives the RSSIs and identifierinformation for only two mobile devices. The task of the controller isto match each identifier with the proper mobile device, that is, withthe appropriate one of the individuals detected in the image.

From the received signal strength indicators, the controller can computeapproximate distances to the RF-transmitting sources (i.e., the mobiledevices). The greater the number of RF nodes that provide RSSI values,the more precisely the controller can estimate the location of eachmobile device with respect to those RF nodes (e.g., throughtriangulation or multilateration). Alternatively, the controller candeduce a closest-to-farthest order for the mobile devices based on theircorresponding RSSI values, without computing relative location ordistance estimates. The controller arranges (step 612) theRF-transmitting mobile devices in distance order with respect to the RFnode 120-1 and with respect to the RF node 120-2.

Using the known locations of the RF nodes 120-1, 120-2 and the camera220 and camera angles, the controller compares (step 614) the depthorder of the persons detected in the image with the distance order(s)determined for the mobile devices based on the signal strengthindicators provided by the RF nodes 120. Based on the comparison, thecontroller matches (step 616) identifiers with the detected individuals.

For example, based on the example locations of the individuals 500 inFIG. 5 , the RF node 120-1 determines that the signal strength of themobile device 140-1 is greater than that of the mobile device 140-2,whereas the RF node 120-2 finds the opposite is the case. For example,the controller may compute from the RSSI values that the mobile device140-1 is approximately 6 feet away from the RF node 120-1 andapproximately 9 feet away from the RF node 120-2 and that the mobiledevice 140-2 is approximately 11 feet away from the RF node 120-1 andapproximately 4 feet away from the RF nodes 120-2. Accordingly, from thecomputed estimated distances, a distance order for mobile devices withrespect to each RF node emerges; the controller 150 thus finds that themobile device 140-1 is closer to the RF node 120-1 than is the mobiledevice 140-2.

Further, in the captured image, the controller detects three people andcan determine a relative distance from the camera of each person. Depthinformation associated with the image can help with this relativedistance determination. Because the locations of the RF nodes 120-1,120-2 relative to each camera and to each other are known, thecontroller can then provisionally assign identifiers to individuals(i.e., presumably, the mobile device carrying persons). The RSSI values,any signal-strength-based distance calculations, distance orders formobile devices, and the depth information associated with humansdetected in the image, guide these assignments. Other information canfurther enhance the decision-making process. For instance, the entrancemay be sixteen feet in width, which provides an approximate upper boundfor possible distances of people from the camera and from the RF nodesand helps establish the locations of persons passing through entrance.The controller can also consider detected gaps in the image betweenadjacent individuals. Other indicators, such as orientation markersdeliberately placed and calibrated within the camera(s)′ field of view,such as on a wall or shelf that is above or on either side of theentrance, can assist the determination. Fixed features inherent to theenterprise, examples of which include but are not limited to a pillar,beam, airduct, ceiling tile, pipe, window, door, shelf, and counter, canserve as orientation markers. As another example, the RF node(s) canfall within the camera's field of view at the entrance, and the cameracan thus capture the RF node in the same image that captures theplurality of individuals 500. During the image processing, thecontroller can then use the RF node's location in the image as anorientation point to help determine the location each detected humanrelative to the RF node.

Further image processing of images captured by the camera can also beemployed to resolve any ambiguity. For instance, the controller canattempt to detect mobile devices in the hands of the detectedindividuals. Detecting a cell phone in one's possession, however, is notconclusive, as the phone may be turned off or it may not becommunicating with the personalized navigation system. In addition, notdetecting the cellphone in one's possession is likewise inconclusive,for an operating RF-transmitting mobile device phone may be concealed ina person's handbag or pocket. Such information, notwithstanding itsinconclusive nature, can be used to increase the confidence level of acorrect matching of mobile phones to humans detected in an image.

Based on all its available information, the controller can assign, forexample, the mobile device closest to the RF node 120-1 to the person500-1 in the image closest to the camera, and the mobile device fartherfrom the RF node 120-1 to the person 500-2 next in the image closest tocamera. If ambiguity remains (i.e., a confidence level for correctmatching does not exceed a threshold), for example, because not allindividuals are carrying RF-transmitting mobile devices, such asindividual 500-3, or because individuals are walking close to eachother, the controller may collect (step 618) additional RSSI informationfrom multiple RF nodes 120-1, 120-2 to determine 2-dimensional rangeinformation to each of the mobile phones. Again, the greater the numberof RF nodes, the more accurate this range determination. Thisapproximate directional information can be used to calibrate thelocation of the smartphone-toting individuals within the captured imagesand distinguish them from an individual who is not carrying asmartphone.

The RSSI measurements from a single RF node (e.g., 120-1) may be enoughfor the controller to determine relative distance among the mobiledevices from that RF node and find the proper match to a human detectedin the image. Because the distances from the camera to the RF node, fromthe RF Node to each mobile device based on RSSI values, and from thecamera to each detected human based on depth information, are pre-knownor computed, the controller can compute approximate locations of eachmobile device and match them to the humans detected in the image. Aspreviously described, other visual orientation markers and/or subsequentvisual and RF tracking (via subsequent RSSI information) can aid in thedetermination.

FIG. 7 shows an embodiment of another process 700 for distinguishingamong multiple persons located at the entrance of the businessenterprise. At least one of the people is carrying and operating anRF-transmitting mobile device that runs the aforementioned applicationsoftware (either as a foreground or background service) to communicatewith the RF nodes 120 of the business enterprise. For purposes ofillustration, consider that the individuals 500-1, 500-2, and 500-3(FIG. 5 ) are near the entrance. The mobile devices 140-1, 140-2 of theindividuals 500-1, 500-2, respectively, communicate (step 702) with theRF nodes 120-1, 120-2, sending them their identifiers, and, at about thesame time or momentarily after, the camera 220 captures (step 704) animage with all three individuals in it.

For this process, each RF node 120 has an antenna array and estimates(step 706) an angle of arrival for the radio signal coming from eachmobile device 140 and associates that angle of arrival measurement withthe corresponding identifier. Each RF node 120 can determine thedirection of a RF signal incident on the antenna array by measuring thetime difference of arrival (TDOA) of the RF signal at individual antennaof the antenna array. An RF node can make this TDOA measurement bymeasuring phase difference in the RF signal received by each antenna orantenna element in the antenna array. From these time differences, theRF node can calculate the RF signal's angle of arrival. As used herein,this angle of arrival corresponds to an angular position or angle linefor the RF signal source, namely, the mobile device 140. Having angularposition data for a given mobile device at two (or more) RF nodes 120enables computation of an intersection point that corresponds to thelocation of that mobile device. Approaches other than TDOA forcalculating angle of arrival are known in the art and may be usedwithout departing from the principles described herein.

At step 708, the RF nodes send the calculated angle of arrival (i.e.,angular position) measurements and their associated identifiers, and thecamera sends the captured image, to the controller. The controllerassociates the captured image with the angular position measurements andassociated identifiers because of the near synchronicity of when thecamera captured the image and the RF nodes received the RF signals. Thecontroller also knows the relative locations of the RF nodes withrespect to each other and to the camera and tracks which RF node sentwhich angular position measurements.

At step 710, the controller detects the three individuals 500-1, 500-2,500-3 in the image and determines an angular alignment and/or depthorder of the detected individuals with respect to the camera. Theangular alignment corresponds to a left-to-right or right-to-leftappearance of the individuals in the image. For example, from theperspective of the camera, individual 500-2 is between individuals 500-1and 500-3 in the image, with individual 500-3 being on the left ofindividual 500-2 and individual 500-1 on the right of individual 500-2.The determination of depth order may use depth information obtained bythe camera or by a depth sensor calibrated to the camera.

The controller can rearrange the angular alignment and/or depth order ofhumans in the image to be from the perspective of or in respect to eachRF node. This rearrangement facilitates comparing the angular alignmentof individuals in the image with the angular alignment of the mobiledevices according to the angular positions determined by that RF node.

In this example, the controller receives the angular positions andidentifier information for only two of the three mobile devices 140-1,140-2 (FIG. 5 ) captured in the image. The task of the controller is tomatch each identifier with the proper mobile device, that is, with theappropriate one of the individuals detected in the image.

From the received or computed angular positions, the controller cancompute approximate locations of the RF-transmitting sources (i.e., themobile devices). The greater the number of RF nodes that provide angleof arrival or angular positions, the more precisely the controller canestimate the locations of each mobile device with respect to those RFnodes (e.g., through intersection of angle lines from two RF nodes orthrough triangulation or multilateration with angle of arrival linesfrom three or more RF nodes).

Alternatively, or in addition, the controller (step 712) can deduce aleft-to-right (or right-to-left) arrangement for the mobile devicesbased on their corresponding angular positions, without computingrelative location or distance estimates. For example, based on theangular positions computed by the RF nodes 120-1, 120-2, the controllerdetermines the mobile device 140-2 to be on the left of mobile device140-1 from the perspective of the RF node 120-1 and determines themobile device 140-2 to be on the right of mobile device 140-1 from theperspective of the RF node 120-2.

Using the known locations of the RF nodes 120-1, 120-2 and the camera220 and camera angles, the controller can compare (step 714) the angularalignment of the persons detected in the image with the angulararrangement or order determined for the mobile devices based on theangular positions provided by the RF nodes 120. Based on the comparison,the controller matches (step 716) identifiers with the detectedindividuals. For example, the controller can assign the mobile device140-1 on the right according to the RF node 120-1 to the rightmostperson 500-1 in the image, and the mobile device 140-2 on the leftaccording to the RF node 120-1 to the person 500-2 left of person 500-1in the image. This matching can operate without ambiguity when thenumber of mobile phones 140 matches the number of persons detected inthe image (e.g., if individual 500-3 were not in the captured image). Itcan also be performed using the angular order determined for the mobiledevices based on the angular positions provided by only one of the RFnodes 120, although using more than one RF node improves precision.

Alternatively, or in addition, the controller can compute the locationof each mobile device based on the angular positions provided by the RFnodes 120-1, 120-2 (e.g., by computing an intersection of the two anglelines). This location has a depth component (from a reference point,e.g., the camera). The controller can rank the mobile devices in a depthorder based on the depth components of the locations computed for themobile devices. The controller can then match this depth order to adepth order of the individuals in the captured image.

As an aid to the correct matching of mobile devices to individuals (orindividuals to mobile devices), particularly when individuals, such asindividual 500-3, who are not using a mobile device may cause ambiguity,the controller can collect (step 718) RSSI information from multiple RFnodes 120-1, 120-2, to determine 2-dimensional range information to eachof the mobile phones and to match individuals to mobile devices aspreviously described. Thus, the controller can use the signal strengthinformation to supplement the matching based on angle of arrivalcomputations. Again, the greater the number of RF nodes, the moreaccurate this RSSI-based range determination. This approximate distanceinformation can also be used to calibrate the location of thesmartphone-toting individuals within the captured images and distinguishthem from an individual who is not carrying a smartphone (e.g.,individual 500-3).

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method, and computer programproduct. Thus, aspects of the present invention may be embodied entirelyin hardware, entirely in software (including, but not limited to,firmware, program code, resident software, microcode), or in acombination of hardware and software. In addition, aspects of thepresent invention may be in the form of a computer program productembodied in one or more computer readable media having computer readableprogram code stored thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. The computer readablemedium may be a non-transitory computer readable storage medium,examples of which include, but are not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination thereof.

As used herein, a computer readable storage medium may be any tangiblemedium that can contain or store a program for use by or in connectionwith an instruction execution system, apparatus, device, computer,computing system, computer system, or any programmable machine or devicethat inputs, processes, and outputs instructions, commands, or data. Anon-exhaustive list of specific examples of a computer readable storagemedium include an electrical connection having one or more wires, aportable computer diskette, a floppy disk, a hard disk, a random accessmemory (RAM), a read-only memory (ROM), a USB flash drive, annon-volatile RAM (NVRAM or NOVRAM), an erasable programmable read-onlymemory (EPROM or Flash memory), a flash memory card, an electricallyerasable programmable read-only memory (EEPROM), an optical fiber, aportable compact disc read-only memory (CD-ROM), a DVD-ROM, an opticalstorage device, a magnetic storage device, or any suitable combinationthereof.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. As used herein, acomputer readable storage medium is not a computer readable propagatingsignal medium or a propagated signal.

Program code may be embodied as computer-readable instructions stored onor in a computer readable storage medium as, for example, source code,object code, interpretive code, executable code, or combinationsthereof. Any standard or proprietary, programming or interpretivelanguage can be used to produce the computer-executable instructions.Examples of such languages include C, C++, Pascal, JAVA, BASIC,Smalltalk, Visual Basic, and Visual C++.

Transmission of program code embodied on a computer readable medium canoccur using any appropriate medium including, but not limited to,wireless, wired, optical fiber cable, radio frequency (RF), or anysuitable combination thereof.

The program code may execute entirely on a user's device, such as themobile device 140, partly on the user's device, as a stand-alonesoftware package, partly on the user's device and partly on a remotecomputer or entirely on a remote computer or server. Any such remotecomputer may be connected to the user's device through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, through the Internet, using an Internet Service Provider).

Additionally, the methods of this invention can be implemented on aspecial purpose computer, a programmed microprocessor or microcontrollerand peripheral integrated circuit element(s), an ASIC or otherintegrated circuit, a digital signal processor, a hard-wired electronicor logic circuit such as discrete element circuit, a programmable logicdevice such as PLD, PLA, FPGA, PAL, or the like. In general, any devicecapable of implementing a state machine that is in turn capable ofimplementing the proposed methods herein can be used to implement theprinciples of this invention.

Furthermore, the disclosed methods may be readily implemented insoftware using object or object-oriented software developmentenvironments that provide portable source code that can be used on avariety of computer or workstation platforms. Alternatively, thedisclosed system may be implemented partially or fully in hardware usingstandard logic circuits or a VLSI design. Whether software or hardwareis used to implement the systems in accordance with this invention isdependent on the speed and/or efficiency requirements of the system, theparticular function, and the particular software or hardware systems ormicroprocessor or microcomputer systems being utilized. The methodsillustrated herein however can be readily implemented in hardware and/orsoftware using any known or later developed systems or structures,devices and/or software by those of ordinary skill in the applicable artfrom the functional description provided herein and with a general basicknowledge of the computer and image processing arts.

Moreover, the disclosed methods may be readily implemented in softwareexecuted on programmed general-purpose computer, a special purposecomputer, a microprocessor, or the like. In these instances, the systemsand methods of this invention may be implemented as program embedded onpersonal computer such as JAVA® or CGI script, as a resource residing ona server or graphics workstation, as a plug-in, or the like. The systemmay also be implemented by physically incorporating the system andmethod into a software and/or hardware system.

Having described above several aspects of at least one embodiment, it isto be appreciated various alterations, modifications, and improvementswill readily occur to those skilled in the art. Such alterations,modifications, and improvements are intended to be part of thisdisclosure and are intended to be within the scope of the invention.Embodiments of the methods and apparatuses discussed herein are notlimited in application to the details of construction and thearrangement of components set forth in the foregoing description orillustrated in the accompanying drawings. The methods and apparatusesare capable of implementation in other embodiments and of beingpracticed or of being carried out in various ways. Examples of specificimplementations are provided herein for illustrative purposes only andare not intended to be limiting. Also, the phraseology and terminologyused herein is for the purpose of description and should not be regardedas limiting. The use herein of “including,” “comprising,” “having,”“containing,” “involving,” and variations thereof is meant to encompassthe items listed thereafter and equivalents thereof as well asadditional items. References to “or” may be construed as inclusive sothat any terms described using “or” may indicate any of a single, morethan one, and all the described terms. Any references to front and back,left and right, top and bottom, upper and lower, and vertical andhorizontal are intended for convenience of description, not to limit thepresent systems and methods or their components to any one positional orspatial orientation. Accordingly, the foregoing description and drawingsare by way of example only, and the scope of the invention should bedetermined from proper construction of the appended claims, and theirequivalents.

What is claimed is:
 1. A method of tracking persons in a building; themethod comprising the steps of: receiving, in electromagnetic signalstransmitted by a mobile device carried by a person, an identifierassociated with the person; capturing an image in synchronicity with orin near synchronicity with receiving the identifier associated with theperson in the electromagnetic signals transmitted by the mobile device;image processing the captured image to detect an individual in the imageand to acquire object information related to the individual; assigningthe individual detected in the image to the person associated with theidentifier in response to the synchronicity or near synchronicity of theimage being captured with receipt of the electromagnetic signals havingthe identifier; and optically tracking the person throughout thebuilding based on the acquired object information related to theindividual.
 2. The method of claim 1, wherein the step of assigning theindividual detected in the image to the person associated with theidentifier includes: matching an angle of the individual within theimage with respect to an optical device that captured the image with anangle of arrival of the electromagnetic signals transmitted by themobile device as measured by at least one radiofrequency (RF) node thatreceived the electromagnetic signals, said matching taking intoconsideration a known location of the at least one RF node relative to aknown location of the optical device.
 3. The method of claim 1, Whereinthe step of assigning the individual detected in the image to the personassociated with the identifier includes: matching a distance of theindividual in the image from an optical device that captured the imagewith a distance of the mobile device from at least one radiofrequency(RF) node that received the electromagnetic signals based on a receivedsignal strength indicator (RSSI) value of the electromagnetic signals asmeasured by that at least one RF node, said matching taking intoconsideration a known location of the at least one RF node relative to aknown location of the optical device.
 4. The method of claim 1, whereinthe step of assigning the individual detected in the image to the personassociated with the identifier includes: matching a direction of theindividual within the image with respect to an optical device thatcaptured the image with a direction of the mobile device with respect toat least one radiofrequency (RF) node that is based on time differencesof arrival (TDOA) of the electromagnetic signals as measured by the atleast one RF node, said matching taking into consideration a knownlocation of the at least one RF node relative to a known location of theoptical device.
 5. The method of claim 1, wherein the step of assigningthe individual detected in the image to the person associated with theidentifier includes: detecting in the image a fixed feature of thebuilding that has a location that is calibrated to an optical devicethat captured the image; and comparing a location of the individual inthe image with the location of the fixed fixture.
 6. The method of claim1, wherein the step of assigning the individual detected in the image tothe person associated with the identifier includes: detecting in theimage at least one RF node that receives the electromagnetic signalstransmitted by the mobile device, the at least one RF node having aknown location relative to an optical device that captured the image;and comparing a location of the individual in the image with the knownlocation of the at least one RF node.
 7. The method of claim 1, Whereinthe object information includes an appearance attribute of the person.8. The method of claim 1, further comprising the step of supplementingthe optical tracking of the person throughout the building usingRF-based positioning that is enhanced by inertial data transmitted inthe electromagnetic signals by the mobile device.
 9. A method oftracking persons in a building, the method comprising the steps of:acquiring identification information associated with a person fromelectromagnetic signals transmitted by a mobile device carried by theperson; assigning the identification information associated with theperson to a human detected in an image captured by an optical device;computing a location of the person in the building based on theelectromagnetic signals transmitted by the mobile device carried by theperson; enhancing the computed location of the person in the buildingbased on inertial data included in the electromagnetic signalstransmitted by the mobile device carried by the person; determining acurrent position of the person in the building based on the imagecaptured of the human; and tracking the person through the building bysupplementing the current position of the person in the building that isdetermined based on the image captured of the human with the computedlocation of the person in the building that is enhanced by the inertialdata included in the electromagnetic signals transmitted by the mobiledevice carried by the person.
 10. The method of claim 9, wherein thestep of assigning the identification information associated with theperson to a human detected in an image captured by an optical deviceincludes matching an angle of the human within the image with respect tothe optical device with an angle of arrival of the electromagneticsignals transmitted by the mobile device as measured by at least oneradiofrequency (RF) node that received the electromagnetic signals, saidmatching taking into consideration a known location of the at least oneRF node relative to a known location of the optical device.
 11. Themethod of claim 9, wherein the step of assigning the identificationinformation associated with the person to a human detected in an imagecaptured by an optical device includes matching a distance of the humanin the image from the optical device with a distance of the mobiledevice from at least one radiofrequency (RF) node that received theelectromagnetic signals based on a received signal strength indicator(RSSI) value of the electromagnetic signals as measured by the at leastone RF node, said matching taking into consideration a known location ofthe at least one RF node relative to a known location of the opticaldevice.
 12. The method of claim 9, wherein the step of assigning theidentification information associated with the person to a humandetected in an image captured by an optical device includes matching adirection of the human within the image with respect to an opticaldevice with a direction of the mobile device with respect to at leastone radiofrequency (RF) node that received the electromagnetic signalsbased on time differences of arrival (TDOA) of the electromagneticsignals as measured by the at least one RF node, said matching takinginto consideration a known location of the at least one RF node relativeto a known location of the optical device.
 13. The method of claim 9,wherein the step of assigning the identification information associatedwith the person to a human detected in an image captured by an opticaldevice includes: detecting in the image a fixed feature of the buildingthat has a known location relative to the optical device; and comparinga location of the human in the image with the known location of thefixed fixture.
 14. The method of claim 9, wherein the step of assigningthe identification information associated with the person to a humandetected in an image captured by an optical device includes: detectingin the image at least one RF node that receives the electromagneticsignals transmitted by the mobile device, the at least one RF node beingdisposed at a known location relative to the optical device; andcomparing a location of the human detected in the image with the knownlocation of the at least one RF node.
 15. A system for tracking personsin a building, the system comprising: at least one radiofrequency (RF)node each with one or more receiver antennas receiving identificationinformation associated with a person in electromagnetic signalstransmitted by a mobile device carried by the person; at least oneoptical device capturing an image of a human in the building; and acontroller in communication with the at least one RF node, to receivethe identification information therefrom, and with the at least oneoptical device to receive the captured image therefrom, the controllerbeing configured to: assign the identification information associatedwith the person to the human detected in an image captured by the atleast one optical device; compute a location of the person in thebuilding based on the electromagnetic signals transmitted by the mobiledevice carried by the person; enhance the computed location of theperson in the building based on inertial data included in theelectromagnetic signals transmitted by the mobile device carried by theperson; determine a current position of the person in the building basedon the image captured of the human; and track the person through thebuilding by supplementing the current position of the person in thebuilding that is determined based on the image captured of the humanwith the computed location of the person in the building that isenhanced by the inertial data included in the electromagnetic signalstransmitted by the mobile device carried by the person.
 16. The systemof claim 15, wherein the controller is configured to assign theidentification information associated with the person to a humandetected in an image captured by the at least one optical device bymatching an angle of the human within the image with respect to the atleast one optical device with an angle of arrival of the electromagneticsignals transmitted by the mobile device as measured by the at least oneRF node that received the electromagnetic signals, said matching takinginto consideration a known location of the at least one RF node relativeto a known location of the at least one optical device.
 17. The systemof claim 15, wherein the controller is configured to assign theidentification information associated with the person to a humandetected in an image captured by the at least one optical device bymatching a distance of the human in the image from the optical devicewith a distance of the mobile device from at least one RF node that isbased on a received signal strength indicator (RSSI) value of theelectromagnetic signals as measured by the at least one RF node, saidmatching taking into consideration a known location of the at least oneRF node relative to a known location of the at least one optical device.18. The system of claim 15; wherein the controller is configured toassign the identification information associated with the person to ahuman detected in an image captured by the at least one optical deviceby: matching a direction of the human within the image with respect tothe at least one optical device with a direction of the mobile devicewith respect to the at least one RF node that is based on timedifferences of arrival (TDOA) of the electromagnetic signals as measuredby the at least one RF node, said matching taking into consideration aknown location of the at least one RF node relative to a known locationof the at least one optical device.
 19. The system of claim 15, whereinthe controller is configured to assign the identification informationassociated with the person to a human detected in an image captured bythe at least one optical device by: detecting in the image a fixedfeature of the building that has a known location relative to the atleast one optical device; and comparing a location of the human in theimage with the known location of the fixed fixture.
 20. The system ofclaim 15; wherein the controller is configured to assign theidentification information associated with the person to a humandetected in an image captured by the at least one optical device by:detecting in the image the at least one RF node that receives theelectromagnetic signals transmitted by the mobile device, the at leastone RF node being disposed at a known location relative to the at leastone optical device; and comparing a location of the human detected inthe image with the known location of the at least one RF node.